Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[MXNET-432] Add Foreach #11531

Merged
merged 135 commits into from
Jul 2, 2018
Merged

[MXNET-432] Add Foreach #11531

merged 135 commits into from
Jul 2, 2018

Conversation

zheng-da
Copy link
Contributor

@zheng-da zheng-da commented Jul 2, 2018 •
edited
Loading

Description

This PR is a copy of #10451.

This PR reconstructs MXNet IR system to support control flow operators and adds one control flow operator: foreach. It takes a Python function as input and run the function over the elements in the input array. foreach is similar to scan in TensorFlow.

This PR is part of the proposal of adding a set of control flow operators to MXNet.
https://cwiki.apache.org/confluence/display/MXNET/Optimize+dynamic+neural+network+models+with+control+flow+operators

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

@zheng-da
Test input a graph.
e86d92c
@zheng-da
Update foreach to execute the subgraph.
7eacfab
@zheng-da
print inputs/outputs in foreach.
6874f7d
@zheng-da
Remove print.
8f5e62e
@zheng-da
add test code for foreach.
1dd35a8
@zheng-da
exec foreach outside the engine.
1f117cb
@zheng-da
Implements forward of foreach.
cc29fc1
@zheng-da
Add support for variable numbers of inputs and outputs.
036aada
@zheng-da
Add a python wrapper for foreach.
84e0e24
@zheng-da
Fix the order of inputs.
f2a28f0
@zheng-da
hide C version of foreach.
1c4cf0e
@zheng-da
fix a bug temporarily.
9aa896d
@zheng-da
add test with lstm.
6784408
@zheng-da
Test free variables.
f488647
@zheng-da
change for the new interface of InputGraph attribute.
d9b0c50
@zheng-da
Add attribute to the subgraph.
1188d97
@zheng-da
Handle free variables.
c0cd6ac
@zheng-da
Get all input symbols of a subgraph.
74d280b
@zheng-da
Fix shape, dtype and storage inference.
2bc80e3
@zheng-da
reorganize the output of foreach.
68faa17
@zheng-da
Add a gluon RNN unroll with symbol foreach.
3751ca7
@zheng-da
print unnecessary print.
b98e06d
@zheng-da
have imperative and symbolic foreach.
fc575fe
@zheng-da
Fix an error after moving foreach.
37da6fb
@zheng-da
Fix imperative foreach
f41235c
@zheng-da
Fix a minor problem.
214c1c2
@zheng-da
Use CachedOp to execute subgraph.
9aabc74
@zheng-da
update TODO.
7fc0155
@zheng-da
make foreach op use FStatefulComputeEx.
0d3613a 
TODO we need to change stateful executor to handle subgraph.
@zheng-da
Add backward.
f33d0f4
@zheng-da
fix attribute name.
64e4ff6
@zheng-da
move benchmark.
d243c12
@zheng-da
fix the mapping of operator inputs/outputs and subgraph inputs/outputs.
3afc4d4
@zheng-da
add tests for dtype/shape inference.
62901fe
@zheng-da
reorganize tests.
14b8fb9
@zheng-da
fix a bug of cutting NodeEntry.
f7d7f17 
When two node entries refer to the same output of a node, we should
create only one var node for these two node entries.
@zheng-da
fix lint error.
7d012d9
@zheng-da
handle the case that outputs are inputs.
b83253d
@zheng-da
handle the case that inputs aren't used.
275bbf1
@zheng-da
handle the case without output data.
0e6df9a
@zheng-da
fix a bug in foreach backward.
dfadc8d
@zheng-da
fix a bug when there isn't output data.
5e9cf5f
@zheng-da
Fix lint error.
696f53c
@zheng-da
test diff Gluon RNN cells.
094977d
@zheng-da
test all symbol RNN cells.
7b016ae
@zheng-da
adjust the test precision.
9609ce8
@zheng-da
Fix a bug in getting a list of variable names.
fa8abbd 
We can't get a list of variable names from a hashtable. The order can't
be guaranteed. Python2 and Python3 output different orders.
@zheng-da
fix lint error.
8e74d80
@zheng-da
Test 1D array.
2439105
@zheng-da
fix a bug when subgraph inputs and outputs share NDArray.
53cdbfa
@zheng-da
fix.
9bff317
@zheng-da
fix
d3687ef
@zheng-da
add comments.
392a7e4
@piiswrong piiswrong merged commit 030fbc3 into apache:master Jul 2, 2018
@junrushao junrushao mentioned this pull request Jul 17, 2018
Merged
10 tasks
XinYao1994 pushed a commit to XinYao1994/incubator-mxnet that referenced this pull request Aug 29, 2018
@zheng-da @piiswrong
[MXNET-432] Add Foreach (apache#11531)
a486445 
* Test input a graph.

* Update foreach to execute the subgraph.

* print inputs/outputs in foreach.

* Remove print.

* add test code for foreach.

* exec foreach outside the engine.

* Implements forward of foreach.

* Add support for variable numbers of inputs and outputs.

* Add a python wrapper for foreach.

* Fix the order of inputs.

* add test with lstm.

* hide C version of foreach.

* fix a bug temporarily.

* Test free variables.

* change for the new interface of InputGraph attribute.

* Add attribute to the subgraph.

* Handle free variables.

* Get all input symbols of a subgraph.

* Fix shape, dtype and storage inference.

* reorganize the output of foreach.

* Add a gluon RNN unroll with symbol foreach.

* print unnecessary print.

* have imperative and symbolic foreach.

* Fix an error after moving foreach.

* Fix imperative foreach

* Fix a minor problem.

* Use CachedOp to execute subgraph.

* update TODO.

* make foreach op use FStatefulComputeEx.

TODO we need to change stateful executor to handle subgraph.

* Add backward.

* Fix bugs.

* enable backward test in lstm.

* Fix a bug in foreach backward for free variables.

* change for the new CachedOp.

* Detect the backward computation.

* Fix bugs in foreach.

* fix tests.

* update tests.

* check state shape.

* enable nested foreach.

* remove print.

* fix a bug in test.

* handle infer storage type for backward.

* address comments.

* address comments.

* move some common functions out.

* address comments.

* fix lint.

* Fix lint.

* add doc.

* undo modification in imperative.h

* add doc and remove example code.

* fix lint.

* fix lint.

* Fix lint.

* make nd.foreach and sym.foreach consistent.

* fix compile error.

* address comments.

* update.

* check for loop only works for dense arrays.

* move control flow op out of nn/

* fix include.

* add a test in gluon.

* work for GPU.

* small fix.

* remove subgraph_name

* create loop state for reuse in the future.

* move code.

* Revert "remove subgraph_name"

This reverts commit 977f562.

* cut graph.

* rename new var nodes.

* Fix tests.

* Fix bugs caused by ctypes (apache#29)

* Add save/load json in testcases for foreach (apache#30)

* support subgraph in stateful executor.

* Fix compilation.

* fix a bug when a subgraph has variable nodes.

* Fix a bug of getting symbols.

* copy var nodes.

* Fix getting op states.

* fix lint error.

* address comments.

* fix lint error.

* simplify the execution of subgraph in the main thread.

* fix lint error.

* avoid waiting for computation in each iteration.

* reuse cached op for inference.

* share memory across mini-batches.

* reuse memory.

reuse memory between iterations in inference.
reuse memory between mini-batches in training.

* add tests for multiple batches.

* remove entry.

* add benchmark for foreach.

* benchmark large batch size.

* Fix the benchmark for GPU.

* address comments.

* update shape/dtype/storage inference.

* update contrib API docs.

* support nested foreach.

* use a single CachedOp for all iterations.

* use large dim.

* update benchmark.

* update benchmark.

* update benchmark.

* update benchmark.

* return symbol arrays correctly in MXSymbolCutSubgraph.

* return symbol arrays in MXSymbolGetInputSymbols.

* fix lint error.

* use cachedop to infer storage in backward.

* fix scala API.

* update comments.

* fix scala.

* fix test.

* fix attribute name.

* move benchmark.

* fix the mapping of operator inputs/outputs and subgraph inputs/outputs.

* add tests for dtype/shape inference.

* reorganize tests.

* fix a bug of cutting NodeEntry.

When two node entries refer to the same output of a node, we should
create only one var node for these two node entries.

* fix lint error.

* handle the case that outputs are inputs.

* handle the case that inputs aren't used.

* handle the case without output data.

* fix a bug in foreach backward.

* fix a bug when there isn't output data.

* Fix lint error.

* test diff Gluon RNN cells.

* test all symbol RNN cells.

* adjust the test precision.

* Fix a bug in getting a list of variable names.

We can't get a list of variable names from a hashtable. The order can't
be guaranteed. Python2 and Python3 output different orders.

* fix lint error.

* Test 1D array.

* fix a bug when subgraph inputs and outputs share NDArray.

* fix.

* fix

* add comments.
@zheng-da zheng-da deleted the foreach1 branch September 29, 2018 21:32
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@nswamy nswamy Awaiting requested review from nswamy nswamy is a code owner

@szha szha Awaiting requested review from szha szha is a code owner

@yzhliu yzhliu Awaiting requested review from yzhliu yzhliu is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@zheng-da @piiswrong